{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Abstract:** this notebook give an introduction to `sktime` in-memory data containers and data sets, with associated functionality such as in-memory format validation, conversion, and data set loading.\r\n",
    "\r\n",
    "**Set-up instructions:** on binder, this nootebook should run out-of-the-box.\r\n",
    "\r\n",
    "To run this notebook as intended, ensure that `sktime` with basic dependency requirements is installed in your python environment.\r\n",
    "\r\n",
    "To run this notebook with a local development version of sktime, either uncomment and run the below, or `pip install -e` a local clone of the `sktime` `main` branch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from os import sys\n",
    "# sys.path.append(\"..\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In-memory data representations and data loading\r\n",
    "\r\n",
    "`sktime` provides modules for a number of time series related learning tasks.\r\n",
    "\r\n",
    "These modules use `sktime` specific in-memory (i.e., python workspace) representations for time series and related objects, most importantly individual time series and time series panels. `sktime`'s in-memory representations rely on `pandas` and `numpy`, with additional conventions on the `pandas` and `numpy` object.\r\n",
    "\r\n",
    "Users of `sktime` should be aware of these representations, since presenting the data in an `sktime` compatible representation is usually the first step in using any of the `sktime` modules.\r\n",
    "\r\n",
    "This notebook introduces the data types used in `sktime`, related functionality such as converters and validity checkers, and common workflows for loading and conversion:\r\n",
    "\r\n",
    "**Section 1** introduces in-memory data containers used in `sktime`, with examples.\r\n",
    "\r\n",
    "**Section 2** introduces validity checkers and conversion functionality for in-memory data containers.\r\n",
    "\r\n",
    "**Section 3** introduces common workflows to load data from file formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1: in-memory data containers\r\n",
    "\r\n",
    "This section provides a reference to data containers used for time series and related objets in `sktime`.\r\n",
    "\r\n",
    "Conceptually, `sktime` distinguishes:\r\n",
    "\r\n",
    "* the *data scientific abstract data type* - or short: **scitype** - of a data container, defined by relational and statistical properties of the data being represented and common operations on it - for instance, an abstract \"time series\" or an abstract \"time series panel\", without specifying a particular machine implementation in python\r\n",
    "* the *machine implementation type* - or short: **mtype** - of a data container, which, for a defined *scitype*, specifies the python type and conventions on structure and value of the python in-memory object. For instance, a concrete (mathematical) time series is represented by a concrete `pandas.DataFrame` in `sktime`, subject to certain conventions on the `pandas.DataFrame`. Formally, these conventions form a specific mtype, i.e., a way to represent the (abstract) \"time series\" scitype.\r\n",
    "\r\n",
    "In `sktime`, the same scitype can be implemented by multiple mtypes. For instance, `sktime` allows the user to specify time series as `pandas.DataFrame`, as `pandas.Series`, or as a `numpy.ndarray`. These are different mtypes which are admissible representations of the same scitype, \"time series\". Also, not all mtypes are equally rich in metadata - for instance, `pandas.DataFrame` can store column names, while this is not possible in `numpy.ndarray`.\r\n",
    "\r\n",
    "Both scitypes and mtypes are encoded by strings in `sktime`, for easy reference.\r\n",
    "\r\n",
    "This section introduces the mtypes for the following scitypes:\r\n",
    "* `\"Series\"`, the `sktime` scitype for time series of any kind\r\n",
    "* `\"Panel\"`, the `sktime` scitype for time series panels of any kind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 1.1: Time series - the `\"Series\"` scitype\r\n",
    "\r\n",
    "The major representations of time series in `sktime` are:\r\n",
    "\r\n",
    "* `\"pd.DataFrame\"` - a uni- or multivariate `pandas.DataFrame`, with rows = time points, cols = variables\r\n",
    "* `\"pd.Series\"` - a (univariate) `pandas.Series`, with entries corresponding to different time points\r\n",
    "* `\"np.ndarray\"` - a 2D `numpy.ndarray`, with rows = time points, cols = variables\r\n",
    "\r\n",
    "`pandas` objects must have one of the following `pandas` index types:\r\n",
    "`Int64Index`, `RangeIndex`, `DatetimeIndex`, `PeriodIndex`; if `DatetimeIndex`, the `freq` attribute must be set.\r\n",
    "\r\n",
    "`numpy.ndarray` 2D arrays are interpreted as having an `RangeIndex` on the rows, and generally equivalent to the `pandas.DataFrame` obtained after default coercion using the `pandas.DataFrame` constructor.\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import to retrieve examples\n",
    "from sktime.datatypes import get_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 1.1.1: Time series - the `\"pd.DataFrame\"` mtype\r\n",
    "\r\n",
    "In the `\"pd.DataFrame\"` mtype, time series are represented by an in-memory container `obj: pandas.DataFrame` as follows.\r\n",
    "\r\n",
    "* structure convention: `obj.index` must be monotonous, and one of `Int64Index`, `RangeIndex`, `DatetimeIndex`, `PeriodIndex`.\r\n",
    "* variables: columns of `obj` correspond to different variables\r\n",
    "* variable names: column names `obj.columns`\r\n",
    "* time points: rows of `obj` correspond to different, distinct time points\r\n",
    "* time index: `obj.index` is interpreted as a time index.\r\n",
    "* capabilities: can represent multivariate series; can represent unequally spaced series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of a univariate series in `\"pd.DataFrame\"` representation.\r\n",
    "The single variable has name `\"a\"`, and is observed at four time points 0, 1, 2, 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_examples(mtype=\"pd.DataFrame\", as_scitype=\"Series\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of a bivariate series in `\"pd.DataFrame\"` representation.\r\n",
    "This series has two variables, named `\"a\"` and `\"b\"`. Both are observed at the same four time points 0, 1, 2, 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_examples(mtype=\"pd.DataFrame\", as_scitype=\"Series\")[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 1.1.2: Time series - the `\"pd.Series\"` mtype\r\n",
    "\r\n",
    "In the `\"pd.Series\"` mtype, time series are represented by an in-memory container `obj: pandas.Series` as follows.\r\n",
    "\r\n",
    "* structure convention: `obj.index` must be monotonous, and one of `Int64Index`, `RangeIndex`, `DatetimeIndex`, `PeriodIndex`.\r\n",
    "* variables: there is a single variable, corresponding to the values of `obj`. Only univariate series can be represented.\r\n",
    "* variable names: by default, there is no column name. If needed, a variable name can be provided as `obj.name`.\r\n",
    "* time points: entries of `obj` correspond to different, distinct time points\r\n",
    "* time index: `obj.index` is interpreted as a time index.\r\n",
    "* capabilities: cannot represent multivariate series; can represent unequally spaced series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of a univariate series in `\"pd.Series\"` mtype representation.\r\n",
    "The single variable has name `\"a\"`, and is observed at four time points 0, 1, 2, 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_examples(mtype=\"pd.Series\", as_scitype=\"Series\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 1.1.3: Time series - the `\"np.ndarray\"` mtype\r\n",
    "\r\n",
    "In the `\"np.ndarray\"` mtype, time series are represented by an in-memory container `obj: np.ndarray` as follows.\r\n",
    "\r\n",
    "* structure convention: `obj` must be 2D, i.e., `obj.shape` must have length 2. This is also true for univariate time series.\r\n",
    "* variables: variables correspond to columns of `obj`.\r\n",
    "* variable names: the `\"np.ndarray\"` mtype cannot represent variable names.\r\n",
    "* time points: the rows of `obj` correspond to different, distinct time points. \r\n",
    "* time index: The time index is implicit and by-convention. The `i`-th row (for an integer `i`) is interpreted as an observation at the time point `i`.\r\n",
    "* capabilities: cannot represent multivariate series; cannot represent unequally spaced series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of a univariate series in `\"np.ndarray\"` mtype representation.\r\n",
    "There is a single (unnamed) variable, it is observed at four time points 0, 1, 2, 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_examples(mtype=\"np.ndarray\", as_scitype=\"Series\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of a bivariate series in `\"np.ndarray\"` mtype representation.\r\n",
    "There are two (unnamed) variables, they are both observed at four time points 0, 1, 2, 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_examples(mtype=\"np.ndarray\", as_scitype=\"Series\")[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 1.2: Time series panels - the `\"Panel\"` scitype\r\n",
    "\r\n",
    "The major representations of time series panels in `sktime` are:\r\n",
    "\r\n",
    "* `\"pd-multiindex\"` - a `pandas.DataFrame`, with row multi-index (`instances`, `timepoints`), cols = variables\r\n",
    "* `\"numpy3D\"` - a 3D `np.ndarray`, with axis 0 = instances, axis 1 = variables, axis 2 = time points\r\n",
    "* `\"df-list\"` - a `list` of `pandas.DataFrame`, with list index = instances, data frame rows = time points, data frame cols = variables\r\n",
    "\r\n",
    "These representations are considered primary representations in `sktime` and are core to internal computations.\r\n",
    "\r\n",
    "There are further, minor representations of time series panels in `sktime`:\r\n",
    "\r\n",
    "* `\"nested_univ\"` - a `pandas.DataFrame`, with `pandas.Series` in cells. data frame rows = instances, data frame cols = variables, and series axis = time points\r\n",
    "* `\"numpyflat\"` - a 2D `np.ndarray` with rows = instances, and columns indexed by a pair index of (variables, time points). This format is only being converted to and cannot be converted from (since number of variables and time points may be ambiguous).\r\n",
    "* `\"pd-wide\"` - a `pandas.DataFrame` in wide format: has column multi-index (variables, time points), rows = instances; the \"variables\" index can be omitted for univariate time series\r\n",
    "* `\"pd-long\"` - a `pandas.DataFrame` in long format: has cols `instances`, `timepoints`, `variable`, `value`; entries in `value` are indexed by tuples of values in (`instances`, `timepoints`, `variable`).\r\n",
    "\r\n",
    "The minor representations are currently not fully consolidated in-code and are not discussed further below. Contributions are appreciated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 1.2.1: Time series panels - the `\"pd-multiindex\"` mtype\r\n",
    "\r\n",
    "In the `\"pd-multiindex\"` mtype, time series panels are represented by an in-memory container `obj: pandas.DataFrame` as follows.\r\n",
    "\r\n",
    "* structure convention: `obj.index` must be a pair multi-index of type `(RangeIndex, t)`, where `t` is one of `Int64Index`, `RangeIndex`, `DatetimeIndex`, `PeriodIndex` and monotonous. `obj.index` must have name `(\"instances\", \"timepoints\")`.\r\n",
    "* instances: rows with the same `\"instances\"` index correspond to the same instance; rows with different `\"instances\"` index correspond to different instances.\r\n",
    "* instance index: the first element of pairs in `obj.index` is interpreted as an instance index. \r\n",
    "* variables: columns of `obj` correspond to different variables\r\n",
    "* variable names: column names `obj.columns`\r\n",
    "* time points: rows of `obj` with the same `\"timepoints\"` index correspond correspond to the same time point; rows of `obj` with different `\"timepoints\"` index correspond correspond to the different time points.\r\n",
    "* time index: the second element of pairs in `obj.index` is interpreted as a time index. \r\n",
    "* capabilities: can represent panels of multivariate series; can represent unequally spaced series; can represent panels of unequally supported series; cannot represent panels of series with different sets of variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of a panel of multivariate series in `\"pd-multiindex\"` mtype representation.\r\n",
    "The panel contains three multivariate series, with instance indices 0, 1, 2. All series have two variables with names `\"var_0\"`, `\"var_1\"`. All series are observed at three time points 0, 1, 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_examples(mtype=\"pd-multiindex\", as_scitype=\"Panel\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 1.2.2: Time series panels - the `\"numpy3D\"` mtype\r\n",
    "\r\n",
    "In the `\"numpy3D\"` mtype, time series panels are represented by an in-memory container `obj: np.ndarray` as follows.\r\n",
    "\r\n",
    "* structure convention: `obj` must be 3D, i.e., `obj.shape` must have length 2.\r\n",
    "* instances: instances correspond to axis 0 elements of `obj`.\r\n",
    "* instance index: the instance index is implicit and by-convention. The `i`-th element of axis 0 (for an integer `i`) is interpreted as indicative of observing instance `i`. \r\n",
    "* variables: variables correspond to axis 1 elements of `obj`.\r\n",
    "* variable names: the `\"numpy3D\"` mtype cannot represent variable names.\r\n",
    "* time points: time points correspond to axis 2 elements of `obj`.\r\n",
    "* time index: the time index is implicit and by-convention. The `i`-th elemtn of axis 2 (for an integer `i`) is interpreted as an observation at the time point `i`.\r\n",
    "* capabilities: can represent panels of multivariate series; cannot represent unequally spaced series; cannot represent panels of unequally supported series; cannot represent panels of series with different sets of variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of a panel of multivariate series in `\"numpy3D\"` mtype representation.\r\n",
    "The panel contains three multivariate series, with instance indices 0, 1, 2. All series have two variables (unnamed). All series are observed at three time points 0, 1, 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_examples(mtype=\"numpy3D\", as_scitype=\"Panel\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 1.2.3: Time series panels - the `\"df-list\"` mtype\r\n",
    "\r\n",
    "In the `\"df-list\"` mtype, time series panels are represented by an in-memory container `obj: List[pandas.DataFrame]` as follows.\r\n",
    "\r\n",
    "* structure convention: `obj` must be a list of `pandas.DataFrames`. Individual list elements of `obj` must follow the `\"pd.DataFrame\"` mtype convention for the `\"Series\"` scitype.\r\n",
    "* instances: instances correspond to different list elements of `obj`.\r\n",
    "* instance index: the instance index of an instance is the list index at which it is located in `obj`. That is, the data at `obj[i]` correspond to observations of the instance with index `i`.\r\n",
    "* variables: columns of `obj[i]` correspond to different variables available for instance `i`.\r\n",
    "* variable names: column names `obj[i].columns` are the names of variables available for instance `i`.\r\n",
    "* time points: rows of `obj[i]` correspond to different, distinct time points, at which instance `i` is observed.\r\n",
    "* time index: `obj[i].index` is interpreted as the time index for instance `i`.\r\n",
    "* capabilities: can represent panels of multivariate series; can represent unequally spaced series; can represent panels of unequally supported series; can represent panels of series with different sets of variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of a panel of multivariate series in `\"df-list\"` mtype representation.\r\n",
    "The panel contains three multivariate series, with instance indices 0, 1, 2. All series have two variables with names `\"var_0\"`, `\"var_1\"`. All series are observed at three time points 0, 1, 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_examples(mtype=\"df-list\", as_scitype=\"Panel\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: validity checking and mtype conversion\r\n",
    "\r\n",
    "`sktime`'s `datatypes` module provides users with generic functionality for:\r\n",
    "\r\n",
    "* checking in-memory containers against mtype conventions, with informative error messages that help moving data to the right format\r\n",
    "* converting different mtypes to each other, for a given scitype\r\n",
    "\r\n",
    "In this section, this functionality and intended usage worfklows are presented."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 2.1: Preparing data, checking in-memory containers for validity\r\n",
    "\r\n",
    "`sktime`'s `datatypes` module provides convenient functionality for users to check validity of their in-memory data containers, using the `check_is` and `check_raise` functions. Both functions provide generic validity checking functionality, `check_is` returns metadata and potential issues as return arguments, while `check_raise` directly produces informative error messages in case a container does not comply with a given `mtype`.\r\n",
    "\r\n",
    "A recommended notebook workflow to ensure that a given data container is compliant with `sktime` `mtype` specification is as follows:\r\n",
    "\r\n",
    "1. load the data in an in-memory data container\r\n",
    "2. identify the `scitype`, e.g., is this supposed to be a time series (`Series`) or a panel of time series (`Panel`)\r\n",
    "3. select the target `mtype` (see Section 1 for a list), and attempt to manually reformat the data to comply with the `mtype` specification if it is not already compliant\r\n",
    "4. run `check_raise` on the data container, to check whether it complies with the `mtype` and `scitype`\r\n",
    "5. if an error is raised, repeat 3 and 4 until no error is raised"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 2.1.1: validity checking, example 1 (simple mistake)\r\n",
    "\r\n",
    "Suppose we have the following `numpy.ndarray` representing a univariate time series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "y = np.array([1, 6, 3, 7, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to check compatibility with sktime:\r\n",
    "\r\n",
    "(instruction: uncomment and run the code to see the informative error message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import check_raise\n",
    "\n",
    "# check_raise(y, mtype=\"np.ndarray\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this tells us that `sktime` uses 2D numpy arrays for time series, if the `np.ndarray` mtype is used. While most methods provide convenience functionality to do this coercion automatically, the \"correct\" format would be 2D as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_raise(y.reshape(-1, 1), mtype=\"np.ndarray\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For use in own code or additional metadata, the error message can be obtained using the `check_is` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import check_is\n",
    "\n",
    "check_is(y, mtype=\"np.ndarray\", return_metadata=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and metadata is produced if the argument passes the validity check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_is(y.reshape(-1, 1), mtype=\"np.ndarray\", return_metadata=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: if the name of the mtype is ambiguous and can refer to multiple scitypes, the additional argument `scitype` must be provided. This should not be the case for any common in-memory containers, we mention this for completeness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_is(y, mtype=\"np.ndarray\", scitype=\"Series\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 2.1.2: validity checking, example 2 (non-obvious mistake)\r\n",
    "\r\n",
    "Suppose we have converted our data into a multi-index panel, i.e., we want to have a `Panel` of mtype `pd-multiindex`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "cols = [\"instances\", \"time points\"] + [f\"var_{i}\" for i in range(2)]\n",
    "X = pd.concat(\n",
    "    [\n",
    "        pd.DataFrame([[0, 0, 1, 4], [0, 1, 2, 5], [0, 2, 3, 6]], columns=cols),\n",
    "        pd.DataFrame([[1, 0, 1, 4], [1, 1, 2, 55], [1, 2, 3, 6]], columns=cols),\n",
    "        pd.DataFrame([[2, 0, 1, 42], [2, 1, 2, 5], [2, 2, 3, 6]], columns=cols),\n",
    "    ]\n",
    ").set_index([\"instances\", \"time points\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is not obvious whether `X` satisfies the `pd-multiindex` specification, so let's check:\r\n",
    "\r\n",
    "(instruction: uncomment and run the code to see the informative error message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import check_raise\n",
    "\n",
    "# check_raise(X, mtype=\"pd-multiindex\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The informative error message highlights a typo in one of the multi-index columns, so we do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.index.names = [\"instances\", \"timepoints\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the validity check passes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_raise(X, mtype=\"pd-multiindex\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 2.1.3: inferring the mtype\r\n",
    "\r\n",
    "`sktime` also provides functionality to infer the mtype of an in-memory data container, which is useful in case one is sure that the container is compliant but one has forgotten the exact string, or in a case where one would like to know whether an in-memory container is already in some supported, compliant format. For this, only the scitype needs to be specified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import mtype\r\n",
    "\r\n",
    "mtype(X, as_scitype=\"Panel\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 2.2: conversion between mtypes\r\n",
    "\r\n",
    "`sktime`'s `datatypes` module also offers uninfied conversion functionality between mtypes. This is useful for users as well as for method developers.\r\n",
    "\r\n",
    "The `convert` function requires to specify the mtype to convert from, and the mtype to convert to. The `convert_to` function only requires to specify the mtype to convert to, automatically inferring the mtype of the input if it can be inferred. `convert_to` should be used if the input can have multiple mtypes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Section 2.2.1: simple conversion\r\n",
    "\r\n",
    "Example: converting a `numpy3D` panel of time series to `pd-multiindex` mtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import get_examples\n",
    "\n",
    "X = get_examples(mtype=\"numpy3D\", as_scitype=\"Panel\")[0]\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import convert\n",
    "\n",
    "convert(X, from_type=\"numpy3D\", to_type=\"pd-multiindex\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import convert_to\n",
    "\n",
    "convert_to(X, to_type=\"pd-multiindex\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Section 2.2.2: advanced conversion features\r\n",
    "\r\n",
    "`convert_to` also allows to specify multiple output types. The `to_type` argument can be a list of mtypes. In that case, the input passed through unchanged if its mtype is on the list; if the mtype of the input is not on the list, it is converted to the mtype which is the first element of the list.\r\n",
    "\r\n",
    "Example: converting a panel of time series of to either `\"pd-multiindex\"` or `\"numpy3D\"`. If the input is `\"numpy3D\"`, it remains unchanged. If the input is `\"df-list\"`, it is converted to `\"pd-multiindex\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import get_examples\n",
    "\n",
    "X = get_examples(mtype=\"numpy3D\", as_scitype=\"Panel\")[0]\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import convert_to\n",
    "\n",
    "convert_to(X, to_type=[\"pd-multiindex\", \"numpy3D\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = get_examples(mtype=\"df-list\", as_scitype=\"Panel\")[0]\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "convert_to(X, to_type=[\"pd-multiindex\", \"numpy3D\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Section 2.2.3: inspecting implemented conversions\r\n",
    "\r\n",
    "Currently, conversions are work in progress, and not all possible conversions are available - contributions are welcome.\r\n",
    "To see which conversions are currently implemented for a scitype, use the `_conversions_defined` developer method from the `datatypes._convert` module. This produces a table with a \"1\" if conversion from mtype in row row to mtypw in column is implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes._convert import _conversions_defined\n",
    "\n",
    "_conversions_defined(scitype=\"Panel\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3: loading data sets\r\n",
    "\r\n",
    "`sktime`'s `datasets` module allows to load datasets for testing and benchmarking. This includes:\r\n",
    "\r\n",
    "* example data sets that ship directly with `sktime`\r\n",
    "* downloaders for data sets from common repositories\r\n",
    "\r\n",
    "All data retrieved in this way are in `sktime` compatible in-memory and/or file formats.\r\n",
    "\r\n",
    "Currently, no systematic tagging and registry retrieval for the available data sets is implemented - contributions to this would be very welcome."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 3.1: forecasting data sets\r\n",
    "\r\n",
    "`sktime`'s `datasets` module currently allows to load a the following forecasting example data sets:\r\n",
    "\r\n",
    "| dataset name | loader function | properties |\r\n",
    "|----------|:-------------:|------:|\r\n",
    "| Box/Jenkins airline data | `load_airline` | univariate |\r\n",
    "| Lynx sales data | `load_lynx` | univariate |\r\n",
    "| Shampoo sales data | `load_shampoo_sales` | univariate |\r\n",
    "| Pharmaceutical Benefit Scheme data | `load_PBS_dataset` | univariate |\r\n",
    "| Longley US macroeconomic data | `load_longley` | multivariate |\r\n",
    "| MTS consumption/income data | `load_uschange` | multivariate |\r\n",
    "\r\n",
    "`sktime` currently has no connectors to forecasting data repositories - contributions are much appreciated.\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Forecasting data sets are all of `Series` scitype, they can be univariate or multivariate.\r\n",
    "\r\n",
    "Loaders for univariate data have no arguments, and always return the data in the `\"pd.Series\"` mtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datasets import load_airline\n",
    "\n",
    "load_airline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loaders for multivariate data can be called in two ways:\r\n",
    "\r\n",
    "* without an argument, in which case a multivariate series of `\"pd.DataFrame\"` mtype is returned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datasets import load_longley\n",
    "\n",
    "load_longley()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* with an argument `y_name` that must coincide with one of the column/variable names, in which a pair of series `y`, `X` is returned, with `y` of `\"pd.Series\"` mtype, and `X` of `\"pd.DataFrame\"` mtype - this is convenient for univariate forecasting with exogeneous variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y, X = load_longley(y_name=\"TOTEMP\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 3.2: time series classification data sets\r\n",
    "\r\n",
    "`sktime`'s `datasets` module currently allows to load a the following time series classification example data sets:\r\n",
    "\r\n",
    "| dataset name | loader function | properties |\r\n",
    "|----------|:-------------:|------:|\r\n",
    "| Appliance power consumption data | `load_acsf1` | univariate, equal length/index |\r\n",
    "| Arrowhead shape data | `load_arrow_head` | univariate, equal length/index |\r\n",
    "| Gunpoint motion data | `load_gunpoint` | univariate, equal length/index |\r\n",
    "| Italy power demand data | `load_italy_power_demand` | univariate, equal length/index |\r\n",
    "| Japanese vowels data | `load_japanese_vowels` | univariate, equal length/index |\r\n",
    "| OSUleaf leaf shape data | `load_osuleaf` | univariate, equal length/index |\r\n",
    "| Basic motions data | `load_basic_motions` | multivariate, equal length/index |\r\n",
    "\r\n",
    "Currently, there are no unequal length or unequal index time series classification example data directly in `sktime`.\r\n",
    "\r\n",
    "`sktime` also provides a full interface to the UCR/UEA time series data set archive, via the `load_UCR_UEA_dataset` function.\r\n",
    "The UCR/UEA archive also contains time series classification data sets which are multivariate, or unequal length/index (in either combination).\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 3.2.2: time series classification data sets in `sktime`\r\n",
    "\r\n",
    "Time series classification data sets consists of a panel of time series of `Panel` scitype, together with classification labels, one per time series.\r\n",
    "\r\n",
    "If a loader is invoked with minimal arguments, the data are returned as `\"nested_univ\"` mtype, with labels and series to classify in the same `pd.DataFrame`. Using the `return_X_y=True` argument, the data are returned separated into features `X` and labels `y`, with `X` a `Panel` of `nested_univ` mtype, and `y` and a `sklearn` compatible numpy vector of labels:\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datasets import load_arrow_head\n",
    "\n",
    "X, y = load_arrow_head(return_X_y=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The panel can be converted from `\"nested_univ\"` mtype to other mtype formats, using `datatypes.convert` or `convert_to` (see above):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datatypes import convert_to\n",
    "\n",
    "convert_to(X, to_type=\"pd-multiindex\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data set loaders can be invoked with the `split` parameter to obtain reproducible training and test sets for comparison across studies. If `split=\"train\"`, a pre-defined training set is retrieved; if `split=\"test\"`, a pre-defined test set is retrieved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = load_arrow_head(return_X_y=True, split=\"train\")\n",
    "X_test, y_test = load_arrow_head(return_X_y=True, split=\"test\")\n",
    "# this retrieves training and test X/y for reproducible use in studies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 3.2.3: time series classification data sets from the UCR/UEA time series classification repository\r\n",
    "\r\n",
    "The `load_UCR_UEA_dataset` utility will download datasetes from the UCR/UEA time series classification repository and make them available as in-memory datasets, with the same syntax as `sktime` native data set loaders.\r\n",
    "\r\n",
    "Datasets are indexed by unique string identifiers, which can be inspected on the [repository itself](https://www.timeseriesclassification.com/), or via the register in the `datasets.tsc_dataset_names` module, by property:\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datasets.tsc_dataset_names import univariate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The imported variables are all lists of strings which contain the unique string identifiers of datasets with certain properties, as follows:\r\n",
    "\r\n",
    "| register name | uni-/multivariate | equal/unequal length | with/without missing values |\r\n",
    "|----------|:-------------:|------:|------:|\r\n",
    "| `univariate` | only univariate | both included | both included |\r\n",
    "| `multivariate` | only multivariate | both included | both included |\r\n",
    "| `univariate_equal_length` | only univariate | only equal length | both included |\r\n",
    "| `univariate_variable_length` | only univariate | only unequal length | both included |\r\n",
    "| `univariate_missing_values` | only univariate | both included | only with missing values |\r\n",
    "| `multivariate_equal_length` | only multivariate | only equal length | both included |\r\n",
    "| `multivariate_unequal_length` | only multivariate | only unequal length | both included |\r\n",
    "\r\n",
    "Lookup and retrieval using these lists is, admittedly, a bit inconvenient - contributions to `sktime` to write a lookup functions such as `all_estimators` or `all_tags`, based on capability or property tags attached to datasets would be very much appreciated.\r\n",
    "\r\n",
    "An example list is displayed below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "univariate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loader function `load_UCR_UEA_dataset` behaves exactly as `sktime` data loaders, with an additional argument `name` that should be set to one of the unique identifying strings for the UCR/UEA datasets, for instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sktime.datasets import load_UCR_UEA_dataset\n",
    "\n",
    "X, y = load_UCR_UEA_dataset(name=\"Yoga\", return_X_y=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will download the dataset into a local directory (by default: for a local clone, the `datasets/data` directory in the local repository; for a release install, in the local python environment folder). To change that directory, specify it using the `extract_path` argument of the `load_UCR_UEA_dataset` function."
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "interpreter": {
   "hash": "bc250fec99d1b72e5bb23d9fb06e1f1ac90e860438a1535c061277d2caf5ebfc"
  },
  "kernelspec": {
   "display_name": "Python 3.7.10 64-bit ('sktime': conda)",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": ""
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
